Exposing storage resources with differing capabilities

ABSTRACT

Tracking storage resources includes providing a table containing storage resources along with capabilities and statuses thereof, updating the table in response to a change of status of a storage resource, updating the table in response to a change in capabilities of a storage resource and, in response to an inquiry for a storage resource having a particular capability, searching the table for a storage resource having the particular capability. Tracking storage resources may also include adding an element to the table in response to a new resource being added to the system. The capabilities may include RAID striping, data deduplication, and green operation. The status may be one of: on-line, off-line, and full.

BACKGROUND OF THE INVENTION

1. Technical Field

This application relates to the field of storing data, and moreparticularly to the field of data storage services in a scalable highcapacity system.

2. Description of Related Art

It has been estimated that the amount of digital information created,captured, and replicated in 2006 was 161 exabytes or 161 billiongigabytes, which is about three million times the information in all thebooks ever written. It is predicted that between 2006 and 2010, theinformation added annually to the digital universe will increase morethan six fold from 161 exabytes to 988 exabytes. The type of informationresponsible for this massive growth is rich digital media andunstructured business content. There is also an ongoing conversion fromanalog to digital formats—film to digital image capture, analog todigital voice, and analog to digital TV.

The rich digital media and unstructured business content have uniquecharacteristics and storage requirements that are different thanstructured data types (e.g. database records), for which many of today'sstorage systems were specially designed. Many conventional storagesystems are highly optimized to deliver high performance I/O for smallchunks of data. Furthermore, these systems were designed to supportgigabyte and terabyte sized information stores.

In contrast, rich digital media and unstructured business content havegreater capacity requirements (petabyte versus gigabyte/terabyte sizedsystems), less predictable growth and access patterns, large file sizes,billions and billions of objects, high throughput requirements, singlewriter, multiple reader access patterns, and a need for multi-platformaccessibility. Conventional storage systems have met these needs in partby using specialized hardware platforms to achieve required levels ofperformance and reliability. Unfortunately, the use of specializedhardware results in higher customer prices and may not support volumeeconomics as the capacity demands grow large—a differentiatingcharacteristic of rich digital media and unstructured business content.

Some of the cost issues have been addressed with tiered storage, whichattempts to reduce the capital and operational costs associated withkeeping all information on a single high-cost storage tier. However,tiered storage comes with a complex set of decisions surroundingtechnology, data durability, functionality and even storage vendor.Tiered storage solutions may introduce unrelated platforms,technologies, and software titles having non-zero operational costs andmanagement requirements that become strained as the quantity of dataincreases.

In addition, tiered storage may cause a data replica incoherence whichresults in multiple, disjoint copies of information existing across thetiers of storage. For example, storage management software handling databackup and recovery may make multiple copies of information sets on eachstorage tier (e.g. snapshots, backup sets, etc). Information Life-cycleManagement (ILM) software dealing with information migration from onetier to another may create additional and often overlapping copies ofthe data. Replication software may make an extra copy of the informationset within a particular tier in order to increase performance toaccessing applications. Each of these functions typically runsautonomously from one another. The software may be unable to realizeand/or take advantage of the multiple replicas of the same informationset.

In addition, for large scale unstructured information stores, it may bedifficult to maintain a system and manage the environment as componentsfail. For example, a two petabyte information store may be comprised ofeight thousand 250-gigabyte disk drives. Disk failures should be handledin a different manner in a system of this scale so that the systemcontinues to operate relatively smoothly whenever one or only a few ofthe disk drives fail.

Thus, it would be desirable to provide a storage system that addressesdifficulties associated with high-cost specialized hardware, storagetiering, and failure management.

SUMMARY OF THE INVENTION

According to the system described herein, tracking storage resourcesincludes providing a table containing storage resources along withcapabilities and statuses thereof, updating the table in response to achange of status of a storage resource, updating the table in responseto a change in capabilities of a storage resource and, in response to aninquiry for a storage resource having a particular capability, searchingthe table for a storage resource having the particular capability.Tracking storage resources may also include adding an element to thetable in response to a new resource being added to the system. Thecapabilities may include RAID striping, data deduplication, and greenoperation. The status may be one of: on-line, off-line, and full. Thestorage resources may be disk drives. The disk drives may be managed bydata storage servers that present an OSD interface for the disk drives.The table may be maintained by a resource manager server that receivesinformation about storage resources from other servers.

According further to the system described herein, computer software,provided in a computer-readable storage medium, tracks storageresources. The software includes a table that contains storage resourcesalong with capabilities and statuses thereof, executable code thatupdates the table in response to a change of status of a storageresource, executable code that updates the table in response to a changein capabilities of a storage resource, and executable code that searchesthe table for a storage resource having a particular capability inresponse to an inquiry for a storage resource having the particularcapability. The software may also include executable code that adds anelement to the table in response to a new resource being added to thesystem. The capabilities may include RAID striping, data deduplication,and green operation. The status may be one of: on-line, off-line, andfull. The storage resources may be disk drives. The disk drives may bemanaged by data storage servers that present an OSD interface for thedisk drives. The table may be maintained by a resource manager serverthat receives information about storage resources from other servers.

According further to the system described herein, a resource managerthat manages storage resources for a storage system includes aprocessing device and a computer-readable memory coupled to theprocessing device, the computer-readable memory having a table providedin a data structure and containing storage resources along withcapabilities and statuses thereof, the computer-readable memory alsohaving executable code that updates the table in response to a change ofstatus of a storage resource, executable code that updates the table inresponse to a change in capabilities of a storage resource, andexecutable code that searches the table for a storage resource having aparticular capability in response to an inquiry for a storage resourcehaving the particular capability. The computer-readable memory may alsocontain executable code that adds an element to the table in response toa new resource being added to the system. The capabilities may includeRAID striping, data deduplication, and green operation. The status maybe one of: on-line, off-line, and full. The storage resources may bedisk drives. The disk drives may be managed by data storage servers thatpresent an OSD interface for the disk drives.

According further to the system described herein, a data storage systemincludes a plurality of clients and a plurality of servers coupled tothe clients, where a subset of the servers manage storage resourcesusing a table containing storage resources along with capabilities andstatuses thereof, where the subset updates the table in response to achange of status of a storage resource, updates the table in response toa change in capabilities of a storage resource, and searches the tablefor a storage resource having the particular capability in response toan inquiry for a storage resource having a particular capability. Thesubset of servers may add an element to the table in response to a newresource being added to the system. The storage resources may be diskdrives.

According further to the system described herein, providing informationto a resource manager of a data storage system includes providinginformation to the resource manager in response to a change incapabilities of a storage resource, providing information to theresource manager in response to a change in status of a storageresource, and providing information to the resource manager in responseto adding a new storage resource. The storage resources may be diskdrives.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating servers and clients according to anembodiment of the system described herein.

FIGS. 2A and 2B are diagrams illustrating a client coupled to serversand to other network(s) according to an embodiment of the systemdescribed herein.

FIG. 3 is a diagram illustrating a client having server operationssoftware, client software, and a plurality of interfaces therebetweenaccording to an embodiment of the system described herein.

FIG. 4 is a diagram illustrating a file having a metadata file objectand a plurality of data file objects according to an embodiment of thesystem described herein.

FIG. 5 is a diagram illustrating a metadata file object for a fileaccording to an embodiment of the system described herein.

FIG. 6 is a diagram illustrating an example of a layout storage objecttree for a file according to an embodiment of the system describedherein.

FIG. 7 is a diagram illustrating an example of a layout storage objecttree with multiple maps for a file according to an embodiment of thesystem described herein.

FIG. 8 is a diagram illustrating another example of a layout storageobject tree with multiple maps and replication nodes for a fileaccording to an embodiment of the system described herein.

FIG. 9 is a flowchart illustrating a client obtaining a lease for andoperating on a file according to an embodiment of the system describedherein.

FIG. 10 is a flowchart illustrating a client reading data from a fileaccording to an embodiment of the system described herein.

FIG. 11 is a flowchart illustrating a client writing data to a fileaccording to an embodiment of the system described herein.

FIG. 12 is a flowchart illustrating steps performed by a client inconnection with finding an alternative copy of data according to anembodiment of the system described herein.

FIG. 13 is a flowchart illustrating a client writing to synchronousmirrors for data according to an embodiment of the system describedherein.

FIG. 14 is a flow chart illustrating a client converting file names toobject identifiers according to an embodiment of the system describedherein.

FIG. 15 is a diagram illustrating a client having an application in usermemory address space and a having a VFS, file name services, kernel I/Odrivers, layout manager, and a communication interface in kernel memoryaddress space according to an embodiment of the system described herein.

FIG. 16 is a flow chart illustrating operation of a VFS at a clientaccording to an embodiment of the system described herein.

FIG. 17 is a diagram illustrating a client having an application, filename services, user level I/O drivers, and a layout manager in usermemory address space and having a communication interface in kernelmemory address space according to an embodiment of the system describedherein.

FIG. 18 is a diagram illustrating a client having an application, a filepresentation layer, user level I/O drivers, and a layout manager in usermemory address space and having a VFS and communication interface and akernel memory address space to user memory address space bridge inkernel memory address space according to an embodiment of the systemdescribed herein.

FIG. 19 is a diagram illustrating a client having an application in usermemory address space and having file name services, kernel I/O drivers,a layout manager, and a communication interface in kernel address spaceaccording to an embodiment of the system described herein.

FIG. 20 is a diagram illustrating a client having an application, filename services, user level I/O drivers, and a layout manager in usermemory address space and having a communication interface in kernelmemory address space according to an embodiment of the system describedherein.

FIG. 21 is a diagram illustrating a client having an application, filename services, user level I/O drivers, and a layout manager in usermemory address space and having a communication interface and a kernelmemory address space to user memory address space bridge in kernelmemory address space according to an embodiment of the system describedherein.

FIG. 22 is a diagram illustrating a client having an application in usermemory address space and having a Web Services module, kernel I/Odrivers, a layout manager, and a communication interface in kernelmemory address space according to an embodiment of the system describedherein.

FIG. 23 is a diagram illustrating a client having an application, a WebServices layer, user level I/O drivers, and a layout manager in usermemory address space and having a communication interface in kernelmemory address space according to an embodiment of the system describedherein.

FIG. 24 is a diagram illustrating a client having an application, a WebServices layer, user level I/O drivers, and a layout manager in usermemory address space and having a communication interface and a kernelmemory address space to user memory address space bridge in kernelmemory address space according to an embodiment of the system describedherein.

FIG. 25 is a diagram illustrating a client having a plurality ofapplications, a Web Services layer, file name services, user level I/Odrivers, and a layout manager in user memory address space and having aVFS, a communication interface and a kernel memory address space to usermemory address space bridge in kernel memory address space according toan embodiment of the system described herein.

FIG. 26 is a diagram illustrating different types of servers and a usermanagement interface according to an embodiment of the system describedherein.

FIG. 27 is a flow chart illustrating steps performed in connection withusing security managers servers to obtain credentials for using policymanager servers according to an embodiment of the system describedherein.

FIG. 28 is a diagram illustrating a resource manager table according toan embodiment of the system described herein.

FIG. 29 is a flow chart illustrating steps performed in connection withprocessing resource information to update a resource table according toan embodiment of the system described herein.

FIG. 30 is a flow chart illustrating steps performed in connection withfinding a resource with a desired capability according to an embodimentof the system described herein.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

Referring to FIG. 1, a diagram illustrates servers 102 coupled to aplurality of clients 104-106. Each of the clients 104-106 represents oneor more processing devices that receives file services from the servers102. Each of the clients 104-106 may or may not be independent of otherones of the clients 104-106. One or more of the clients 104-106 may be amultiprocessing/multiuser system and possibly have multiple independentusers. The clients 104-106 are meant to represent any number of clients.

The file services provided by the servers 102 may include data storageand retrieval as well as related operations, such as data mirroring,cloning, etc. The servers 102 may be implemented using a plurality ofservices (and/or interconnected file servers including SAN components)that are provided by interconnected processing and/or storage devices.In an embodiment herein, each of the clients 104-106 may be coupled tothe servers 102 using the Web, possibly in conjunction with local TCP/IPconnections. However, it is possible for one or more of the clients104-106 to be coupled to the servers 102 using any other appropriatecommunication mechanism and/or combinations thereof to provide thefunctionality described herein.

Referring to FIG. 2A, the client 104 is shown as being coupled to theservers 102 and to one or more other network(s). The other network(s)may include a local area network (LAN). Thus, the client 104 may be agateway between the servers 102 and a LAN to which one or more otherdevices (not shown) may also be coupled. The client 104 may act as alocal file server to the one or more other devices coupled to the LAN byproviding data from the servers 102 to the one or more other devices. Ofcourse, it is possible for one or more other clients to simultaneous actas gateways to the same or different other network(s). Generally, forthe discussion herein, reference to a particular one of the clients104-106 may be understood to include reference to any or all of theclients 104-106 coupled to the servers 102 unless otherwise indicated.

Referring to FIG. 2B, a diagram shows the client 104 being coupled tothe servers 102 and one or more other network(s) (e.g., a LAN) in aconfiguration that is different from that shown in FIG. 2A. In theconfiguration of FIG. 2B, a router 108 is coupled between the servers102 and the client 104. The router 108 may be any conventional routerthat may be accessed by the client 104. In the configuration of FIG. 2B,the client 104 uses only a single connection point to both the servers102 and to the other network(s). In the configuration of FIG. 2B, theclient 104 may act as local file server and gateway between the servers102 and one or more other devices (not shown) coupled to the othernetwork(s). Of course, any other appropriate connection configurationsmay be used by any of the client 104-106 coupled to the servers 102and/or to other network(s).

Referring to FIG. 3, the client 104 is shown in more detail havingserver operations software 122, client software 124, and an interfacelayer 125 that includes a plurality of interfaces 126-128 between theserver operations software 122 and the client software 124. The serveroperations software 122 facilitates the exchange of information/databetween the client 104 and the servers 102 to provide the functionalitydescribed herein. The server operations software 122 is described inmore detail elsewhere herein. The client software 124 represents anysoftware that may be run on the client 104, including applicationsoftware, operating system software, Web server software, etc., that isnot part of the server operations software 122 or the interface layer125. As described in more detail elsewhere herein, it is possible tohave the client software 124 interact with the servers 102 throughdifferent ones of the interfaces 126-128 at the same time.

The file services described herein may be implemented by the servers 102using a set of file objects where a file that is accessed by the clientsoftware includes a metadata file object which points to one or moredata file objects that contain the data for the file.

Accessing the file would involve first accessing the metadata fileobject to locate the corresponding data file objects for the file. Doingthis is described in more detail elsewhere herein. Note, however, thatany appropriate file object mechanism may be used for the systemdescribed herein.

Referring to FIG. 4, a file 130 is shown as including a metadata fileobject 132 and a plurality of data file objects. The metadata fileobject 132 contains information that points to each of the data fileobjects 134-136. Accessing the file includes first accessing themetadata file object 132 and then using information therein to locatethe appropriate one or more of the corresponding data file object134-136.

Referring to FIG. 5, the metadata file object 132 is shown in moredetail as including an object attributes section 142 and a LayoutStorage Object (LSO) tree section 144. The object attributes sectioncontains conventional file-type attributes such as owner id, group id,access control list, last modification time, last access time, lastchange time, creation time, file size, and link count. Many of theattributes are self-explanatory.

The last modification time corresponds to the last time that the datafor the data objects 134-136 had been modified while the last changetime corresponds to when the object metadata had last been changed. Thelink count indicates the number of other objects that reference aparticular file (e.g., aliases that point to the same file). In anembodiment herein, a file and its related objects are deleted when thelink count is decremented to zero.

The LSO tree section 144 includes a data structure that includes one ormore maps for mapping the logical space of the file to particular datafile objects. The LSO tree section 144 may also indicate any mirrors forthe data and whether the mirrors are synchronous or asynchronous. LSOtrees and mirrors are described in more detail elsewhere herein.

Referring to FIG. 6, a simple LSO tree 160 is shown as including an LSOroot node 162 and a single map 164. The LSO root node 162 is used toidentify the LSO tree 160 and includes links to one or more map(s) usedin connection with the file corresponding to the LSO tree 160. The map164 maps logical locations within the file to actual data storagelocation. A process that accesses logical storage space of a filerepresented by the LSO tree 160 first uses the LSO root node 162 to findthe map 164 and then uses the map 164 to translate logical addresseswithin the file to an actual data storage locations.

Referring to FIG. 7, an LSO tree 170 is shown as including an LSO rootnode 172 and a plurality of maps 174-176. Each of the maps 174-176 mayrepresent a different range of logical offsets within the filecorresponding to the LSO tree 170. For example, the map 174 maycorrespond to a first range of logical offsets in the file. The map 174may map logical locations in the first range to a first actual storagedevice. The map 175 may correspond to a second range of logical offsetsin the file, different than the first range, which may be mapped to adifferent actual storage device or may be mapped to the same actualstorage device as the map 174. Similarly, the map 176 may correspond toa third range of logical offsets in the file, different than the firstrange and the second range, which may be mapped to a different actualstorage device or may be mapped to the same actual storage device as themap 174 and/or the map 175.

Referring to FIG. 8, an LSO tree 180 is shown as including an LSO rootnode 181 and a pair of replication nodes 182 a, 182 b, which indicatethat the underlying data is to be mirrored (replicated) and whichindicate whether the mirror is synchronous or asynchronous. Synchronousand asynchronous mirrors are discussed in more detail elsewhere herein.The node 182 a has a plurality of children maps 183-185 associatedtherewith while the node 182 b has a plurality of children maps 186-188associated therewith. The replication nodes 182 a, 182 b indicate thatthe data corresponding to the maps 183-185 is a mirror of datacorresponding to the maps 186-188. In some embodiments, the nodes 182 a,182 b may be implemented using a single node 189 to indicatereplication.

A process accessing a file having the LSO tree 180 would traverse thetree 180 and determine that data is mirrored. As discussed in moredetail elsewhere herein, depending upon the type of mirroring, theprocess accessing the LSO tree 180 would either write the data to thechildren of both of the nodes 182 a, 182 b or would provide a message toanother process/server (e.g., the servers 102) that would perform theasynchronous mirroring. Mirroring is discussed in more detail elsewhereherein.

For the system described herein, file objects are accessed by one of theclients 104-106 by first requesting, and obtaining, a lease from theservers 102. The lease corresponds to the file objects for theparticular file being accessed and to the type of access. A lease may befor reading, writing, and/or more some other operation (e.g., changingfile attributes). In an embodiment herein, for objects corresponding toany particular file, the servers 102 may issue only one write lease at atime to any of the clients 104-106 but may issue multiple read leasessimultaneously and may issue read lease(s) at the same time as issuing awrite lease. However, in some embodiments it may be possible to obtain alease for a specified logical range of a file for operations only onthat range. Thus, for example, it may be possible for a first client toobtain lease for writing to a first logical range of a file while asecond client may, independently, obtain a lease for writing to a secondand separate logical range of the same file. The two write leases fordifferent logical ranges may overlap in time without violating thegeneral rule that the system never issues overlapping write leases forthe same data.

The lease provided to the clients 104-106 from the servers 102 includessecurity information (security token) that allows the client appropriateaccess to the data. The security token may expire after a certain amountof time. In an embodiment herein, a client accesses data by providing anappropriate security token for the data as well as clientusers/ownership information. Thus, for example, a user wishing to accessdata would first obtain a lease and then would provide the accessrequest to the servers 102 along with the security token and informationidentifying the owner (client) accessing the data. The servers 102 wouldthen determine whether the access requested by the client waspermissible. After the lease expires (the security token expires), theuser requests the lease again. Data security may be implemented usingconventional data security mechanisms.

After obtaining a lease for accessing a file, a client may then cachethe corresponding metadata, including the LSO tree, into local storageof the client. The client may then use and manipulate the local cachedversion of the metadata and may use the metadata to obtain access to thedata. As described in more detail elsewhere herein, a client does notdirectly modify metadata stored by the servers 102 but, instead, sendsupdate messages to the servers 102 to signal that metadata for a filemay need to be modified by the servers 102.

Referring to FIG. 9, a flowchart 200 illustrates steps performed by aclient in connection with requesting a lease for a file (objectsassociated with a file) for performing operations thereon. Processingbegins at a first step 202 where the client requests the lease for thefile. As discussed in more detail elsewhere herein, a client requestinga lease includes specifying the type of access (e.g., read, write,etc.). Following the step 202 is a test step 204 where it is determinedif the request has been granted. If not, then control transfers from thetest step 204 to a step 206 where processing is performed in connectionwith the lease not being granted to the client. The particularprocessing performed at the step 206 may include, for example, providingan error message to the client process requesting access to the filecorresponding to the lease and/or waiting for an amount of time and thenretrying the request. Note that it is possible that a lease for aparticular file is not available at one time is subsequently availableat another time because, for example, the lease is released by anotherclient in between the first request and the second request. In anyevent, any appropriate processing may be performed at the step 206.Following the step 206, processing is complete.

If it is determined at the test step 204 that the least requested at thestep 202 has been granted, then control transfers from the test step 204to a step 208 where the client performs an operation using the file forwhich the lease was granted. Operations performed at the step 208include reading data and/or writing data. Different types of processingthat may be performed at the step 208 are described in more detailelsewhere herein.

Following the step 208 is a test step 212 where it is determined if theoperations performed at the step 208 require an update. In someinstances, a client may obtain a lease and perform operations that donot affect the file or the underlying file objects. For example, aclient may acquire a lease for reading a file and the operationperformed at the step 208 may include the client reading the file. Insuch a case, no update may be necessary since the file and correspondingfile objects (metadata, data objects, etc.) have not changed. On theother hand, if the client obtains a lease for writing data the file andthe operation performed at the step 208 includes writing data to thefile, then the underlying file objects will have been changed and anupdate message needs to be sent the servers 102. If it is determined atthe test step 212 that an update is necessary, then control passes fromthe test step 212 to a step 214 where an update message is sent by theclient to the servers 102.

Following the step 214, or following the step 212 if no update isnecessary, control passes to a test step 216 where it is determined ifthe client is finished with the file. In some instances, the client mayperform a small number of operations on the file, after which the clientwould be finished with the file at the step 216. In other cases, theclient may be performing a series of operations and may not yet havecompleted all of the operations.

If it is determined at the test step 216 that the client is not finishedwith the file, then control passes from the test step 216 to a test step218 where it is determined if the lease for the file has expired. Notethat a lease may be provided by the servers 102 to the client with aparticular expiration time and/or the associated security token mayexpire. In addition, it may be possible for the servers 102 to recallleases provided to clients under certain circumstances. In either case,the lease may no longer be valid. Accordingly, if it is determined atthe step 218 that the lease has expired (and/or has been recalled by theservers 102), then control passes from the test step 218 back to thestep 202 request the lease again. Otherwise, if the lease has notexpired, then control passes from the test step 218 back to the step 208to perform another iteration.

If it is determined at the test step 216 that the client is finishedwith the file, then control passes from the test step 216 to a step 222where the client releases the lease by sending a message to the servers102 indicating that the client no longer needs the lease. Once theclient releases the lease, it may be available for other clients.Following the step 222, processing is complete.

In an embodiment herein, data file objects may be indicated as havingone of four possible states: current, stale, immutable, or empty. Thecurrent state indicates that the data object is up to date and current.The stale state indicates that the data is not valid but, instead,requires updating, perhaps by some other process. In some instances, thestale state may be used only in connection with mirror copies of data(explained in more detail elsewhere herein). Data may be stale becauseit is a mirror of other data that was recently written but not yetcopied. The immutable state indicates that the corresponding data iswrite protected, perhaps in connection with a previous clone (snapshot)operation. The empty state indicates that no actual storage space hasyet been allocated for the data.

Referring to FIG. 10, a flow chart 240 illustrates steps performed by aclient in connection with performing read operations after obtaining aread lease for a file. Processing begins at a first test step 242 whereit is determined if the data object being read is in the current state.If not, then control transfers from the test step 242 to a step 244where it is determined if the data object being read is in the immutablestate. If it is determined at the step 244 that the data object beingread is in the immutable state or if it is determined at the test step242 that the data object being read is in the current state, thencontrol transfers to a step 246 where the read operation is performed. Aclient reads file data by providing the appropriate data file objectidentifier to the servers 102 as well as providing appropriate securitycredentials. Accordingly, the read operation performed at the step 246includes the client sending an appropriate request to the servers 102and waiting for a result therefrom.

Following the step 246 is a test step 248 where it is determined if theservers 102 have returned a result indicating that the data file objectis unavailable. In some cases, a data file object that is otherwisecurrent or immutable may nevertheless become unavailable. For example,the physical storage space that holds the data file object may becometemporarily disconnected and/or temporarily busy doing some otheroperation. If it is determined at the test step 248 that the data fileobject is available, then control transfers from the test step 248 to atest step 252 where it is determined if the read operation wassuccessful. If so, then control transfers from the test step 252 to astep 254 where the result of the read operation is returned to theprocess at the client that caused the read operation to be performed.The result may include the data that was read and a status indicator.Following the step 254, processing is complete.

If it is determined at the test step 252 that the read operationperformed at the step 246 was not successful, then control transfersfrom the test step 252 to a step 256 where error processing isperformed. The particular error processing performed at the step 256 isimplementation dependent and may include, for example, reporting theerror to a calling process and/or possibly retrying the read operation aspecified number of times. Following the step 256, processing iscomplete.

If it is determined at the test step 244 that the data object being readis not in the immutable state, then control transfers from the test step244 to a test step 258 where it is determined if the data object is inthe stale state. If not, then, by virtue of the test steps 242, 244, 258and process of elimination, the data object is in the empty state. In anembodiment herein, reading a data object in the empty state causes zerosto be returned to the calling process. Accordingly, if it is determinedat the test step 258 that the data object is not in the stale state,then control transfers from the test step 258 to a step 262 where zerosare returned in response to the read operation. Following the step 262,processing is complete.

If it is determined at the test step 258 that the data file object is inthe stale state, or if it is determined at the test step 248 that thedata file object is not available, then control transfers to a test step264 to determine if an alternative version of the data file object isavailable for reading. As discussed in more detail elsewhere herein,there may be multiple versions of the same data file objects that existat the same time due to mirroring. Accordingly, if the data file objectbeing read is in the stale state or otherwise unavailable, it may bepossible to read a mirror copy of the data file object that may be inthe current state. The test performed at the step 264 is described inmore detail elsewhere herein.

If it is determined at the test step 264 that an alternative version ofthe data file object is available, then control transfers from the teststep 264 to a step 266 where the alternative version of the data fileobject is selected for use. Following the step 266, control transfersback to the test step 242 for another iteration with the alternativedata file object.

If it is determined at the test step 264 that an alternative version ofthe data file object is not available, then control transfers from thetest step 264 to a step 268 where the client process waits. In anembodiment herein, it may be desirable to wait for a data file object tobecome current and/or available. Following the step 268, controltransfers back to the step 242 for another iteration. Note that, insteadof waiting at the step 268, processing may proceed from the step 264 tothe step 256 to perform error processing if there is no alternative datafile object available. In other embodiments, it may be possible toperform the step 268 a certain number of times and then, if the datafile object is still unavailable or in the stale state and there is noalternative data file object, then perform the error processing at thestep 256.

Referring to FIG. 11, a flow chart 280 illustrates steps performed by aclient in connection with performing write operations after obtaining awrite lease for a file. Processing begins at a first test step 282 whereit is determined if the data file object to which the write is beingperformed is in the immutable state. If so, then control transfers fromthe step 282 to a step 284 where new actual storage space is allocatedfor the data file object to avoid overwriting the immutable data.Allocating new storage space for a data object may include providing anappropriate request to the servers 102. Following the step 284, controltransfers back to the step 282 to begin the processing for the writeoperation again.

If it is determined at the step 282 that the data file object to whichthe write is being performed is not in the immutable state, then controltransfers from the step 282 to a step 286 where it is determined if thedata file object to which the write is being performed is in the stalestate. If not, then control transfers from the test step 286 to a teststep 288 where it is determined if the data file object to which thewrite is being performed is in the empty state. If so, then controltransfers from the step 288 to the step 284, discussed above, where newphysical storage space is allocated. Following the step 284, controltransfers back to the step 282 to begin the processing for the writeoperation again.

If it is determined at the step 288 that the data file object to whichthe write is being performed is not in the empty state, then controltransfers from the test step 288 to a step 292 where the write operationis performed. Note that the step 292 is reached if the data file objectto which the write operation is being performed is not in the immutablestate, not in the stale state, and not in the empty state (and thus isin the current state). A client writes file data by providing theappropriate data file object location identifier to the servers 102 aswell as providing appropriate security credentials. Accordingly, thewrite operation performed at the step 292 includes the client sending anappropriate request to the servers 102 and waiting for a resulttherefrom.

Following the step 292 is a test step 294 where it is determined if thewrite operation performed at the step 292 was successful. If so, thencontrol transfers from the test step 294 to a test step 296 where it isdetermined if there are synchronous mirrors of the data file object towhich the write is being performed. The test performed at the step 296may include, for example, determining if a parent node of the data fileobject in the file LSO tree indicates replication. If not, then controltransfers from the test step 296 to a step 298 where an update (message)is sent to the servers 102 indicating that the write had been performed.Following the step 298, processing is complete.

If it is determined at the test step 296 that there are synchronousmirrors of the data file object to which the write is being performed,then control passes from the test step 296 to a step 302 where the datathat was written at the step 292 is also written to the synchronousmirror(s). The processing performed at the step 302 is discussed in moredetail elsewhere herein. Following the step 302, control transfers tothe step 298, discussed above, where an update (message) is sent to theservers 102. Following the step 298, processing is complete.

If it is determined at the test step 294 that the write operationperformed at the step 292 was not successful, or if it is determined atthe test step 286 that the data file object to which the write operationis being performed is in the stale state, then control transfers to astep 304 where the data file object to which the write is attempting tobe performed is removed from the client's local copy of the LSO tree. Atthe end of the write operation illustrated by the flow chart 280, theclient may inform the servers 102 (at the step 298) of the difficulty inwriting to the data object so that the servers 102 can take appropriateaction, if necessary.

Following the step 304 is a test step 306 where it is determined if analternative version of the data is available. As discussed in moredetail elsewhere herein, there may be multiple versions of the same datafile objects that exist at the same time due to mirroring. Accordingly,if the data file object to which the write operation is being performedis stale or otherwise cannot be written to, it may be possible to writeto a mirror copy of the data. The test performed at the step 306 is likethe test performed at the step 264 and is described in more detailelsewhere herein. If it is determined at the test step 306 that analternative version of the data corresponding to the data file object isavailable, then control transfers from the test step 306 to a step 308where the alternative version is selected for writing. Following thestep 308, control transfers back to the test step 282 for anotheriteration with the alternative data file object.

If it is determined at the test step 306 that an alternative version ofthe data corresponding to the data file object is not available, thencontrol transfers from the test step 306 to a step 312 to perform errorprocessing if there is no alternative available. The particular errorprocessing performed at the step 312 is implementation dependent and mayinclude, for example, reporting the error to a calling process and/orpossibly retrying the write operation a specified number of times beforereporting the error. Following the step 312, control transfers to thestep 298, discussed above, to send update information to the servers102. Following the step 298, processing is complete.

Referring to FIG. 12, a flow chart 320 illustrates in more detail stepsperformed in connection with the alternative available test step 264 ofFIG. 10 and/or the alternative available test step 306 of FIG. 11.Processing begins at a first test step 322 where it is determined if thefile has any mirror data file objects at all. In some instances, a filemay not use mirrors, in which case there would be no alternative copyavailable. Accordingly, if it is determined at the test step 322 thatthe file does not have any mirror data file objects, then controltransfers from the test step 322 to a step 324 where a value is returnedindicating that no alternative copies are available. Following the step324, processing is complete.

If it is determined at the test step 322 that mirror copies areavailable, then control transfers from the test step 322 to a step 326where a pointer is made to point to a first mirror data file object. Forthe processing discussed herein, a pointer may be used to iteratethrough mirror data file objects to find a useable data file object.Following the step 326 is a test step 328 where it is determined if thepointer is past the end of the list of mirror data file objects (hasiterated through all of the mirror data file objects). If so, thencontrol passes from the test step 328 to the step 324, discussed above,to return a value that indicates that no alternatives are available.

If it is determined at the test step 328 that the pointer is not pastthe end of a list of mirror data file objects, then control transfersfrom the test step 328 to a test step 332 where it is determined if thepointer points to a data file object in a stale state. If so, thencontrol transfers from the test step 332 to a step 334 where the pointeris made to point to the next data file object to be examined. Followingthe step 334, control transfers back to the step 328, discussed above,for another iteration. If it is determined at the test step 332 that thepointer does not point to a data file object in the stale state, thencontrol transfers from the test step 332 to a step 336 where the datafile object that is pointed to by the pointer is returned as analternative data file object that may be used by the calling process.Following the step 336, processing is complete.

Referring to FIG. 13, a flow chart 350 illustrates in more detailoperations performed in connection with the step 302 of the flow chart280 of FIG. 11 where data that has been written is copied to a number ofsynchronous mirrors (mirror data file objects). Processing begins at afirst step 352 where a pointer that is used to iterate through themirror data file objects is set to point the first one of the mirrordata file objects. Following the step 352 is a test step 354 where it isdetermined if the pointer used for iterating through the mirror datafile objects points past the end (i.e., if all of the mirror data fileobjects have been processed). If so, then processing is complete.Otherwise, control transfers from the test step 354 to a test step 356where it is determined if the status of the mirror data file objectpointed to by the pointer indicates that the mirror data file object iscurrent. If not, then control passes from the test step 356 to a teststep 358 where it is determined if the status of the mirror data fileobject pointed to by the pointer indicates that the mirror data fileobject is in the stale state. If so, then control passes from the teststep 358 to a step 362 where the mirror data file object is removed fromthe client's local copy of the LSO tree. In an embodiment herein, asynchronous mirror data file object should not be in a stale state and,if that occurs, it may indicate an error condition. Accordingly,following the step 362 is a step 364 where information about the stalemirror is sent to the servers 102, which may perform recovery processingin response thereto.

Note that if a mirror data file object is neither in the stale state norin the current state, then the mirror data file object is either in theempty state or in the immutable state. In either case, it may benecessary to allocate new space for a data file object to which the datais to be written. Accordingly, if it is determined at the test step 358that the data file object is not in the stale state, then control passesfrom the test step 358 to a step 366 where new space is allocated forthe mirror data file object. Following the step 366 is a step 368 wherethe data that is being copied across synchronous mirror data fileobjects is written to the mirror data file object pointed to by thepointer used to iterate through the mirror data file objects. Note thatthe step 368 may also be reached from the test step 356 if it isdetermined that the mirror data file object is current. Following thestep 368 is a step 372 where the pointer used to iterate through themirror data file objects is made to point to the next one. Note that thestep 372 is also reached following the step 364. Following the step 372,control transfers back to the test step 354 for another iteration.

The system described herein may access file objects using objectidentifiers. In an embodiment herein, each file object that is storedamong the servers 102 may be assigned a unique object identifier thatidentifies each file object and distinguishes each file object fromother file objects in the system. However, many applications use a filenaming structure and/or a hierarchical directory to access files anddata therein. For example, a file name “C:\ABC\DEF\GHI.doc” indicates afile called “GHI.doc” stored in a sub-directory “DEF” that is stored inanother directory “ABC” located on a root volume “C”. A nested directorystructure may be provided by implementing directories as special filesthat are stored in other directories. In the example given above, thesub-directory “DEF” may be implemented as a file stored in the directory“ABC”.

The system described herein may present to applications a conventionalnaming structure and directory hierarchy by translating conventionalfile names into file object identifiers. Such a translation service maybe used by other services in connection with file operations. In anembodiment herein, each directory may include a table that correlatesfile names and sub-directory names with file object identifiers. Thesystem may examine one directory at a time and traverse sub-directoriesuntil a target file is reached.

Referring to FIG. 14, a flow chart 380 illustrates steps performed inconnection with providing a file name translation service (file nameservice) that translates a conventional hierarchical file name into afile object identifier. The file name service may receive a conventionalhierarchical file name as an input and may return an object identifier(or, in some cases, an error). Processing begins at a first step 382where the file name service receives a file name, such as a conventionalhierarchical file name. Following the step 382 is a test step 384 whereit is determined if the syntax of the file name is OK. Checking thesyntax of a hierarchical file name is know and includes, for example,checking that only appropriate characters have been used. If it isdetermined at the test step 384 that the syntax is not OK, then controltransfers from the test step 384 to a step 386 where an error indicator(error message) is returned to the calling process. Following the step386, processing is complete.

If it is determined at the test step 384 that the syntax of the providedname is OK, then control transfers from the test step 384 to a step 388where the root directory is read. In an embodiment herein, all file namepaths begin at a single common root directory used for all file objectsstored in the servers 102. In other embodiments, there may be multipleroot directories where specification of a particular root directory maybe provided by any appropriate means, such as using a volume identifier,specifically selecting a particular root directory, etc.

Following the step 388 is a test step 392 where it is determined if thetarget file (or sub-directory that is part of the file name path) is inthe directory that has been read. If not, then control passes from thetest step 392 to the step 386, discussed above, where an error isreturned. In some embodiments, the file-not-found error that resultsfrom the test at the step 392 may be different from the syntax errorthat results from the test at the step 384.

If it is determined that the target file or a sub-directory that is partof the file name path is in the directory that has just been read, thencontrol passes from the test step 392 to a test step 394 where it isdetermined if the directory that has just been read contains the targetfile (as opposed to containing a sub-directory that is part of the filename path). If so, then control passes from the test step 394 to a step396 where the object identifier of the target file object is returned tothe calling process. Following the step 396, processing is complete.

If it is determined at the test step 394 that the directory that hasjust been read contains a sub-directory that is part of the file namepath, then control transfers from the test step 394 to a step 398 wherethe sub-directory is read so that the sub-directory becomes thedirectory being examined. In effect, processing at the step 398traverses the chain of subdirectories to eventually get to the targetfile. Following the step 398, control transfers back to the step 392,discussed above, for a next iteration.

Referring to FIG. 15, a diagram shows the client 104 as including useraddress memory space and kernel address memory space. In an embodimentherein, user address memory space is memory space that is generally usedby user applications and related processes while kernel address memoryspace is memory space that is generally accessible only by systemprocesses, such as an operating system kernel and related processes. Asdiscussed in more detail herein, it is possible to have differentportions of the system described herein reside and operate in the usermemory space and/or the kernel memory space. In addition, it is possiblefor the client 104 to have multiple different interfaces to access fileobjects at the servers.

In FIG. 15, the client 104 is shown as including an application in theuser memory address space and a virtual file system (VFS), file nameservices, kernel I/O drivers, a layout manager, and a communicationinterface in the kernel memory address space. The VFS is an abstractionlayer on top of a more concrete file system. The purpose of a VFS is toallow client applications to access different types of concrete filesystems in a uniform way. The VFS allows the application running on theclient 104 to access file objects on the servers 102 without theapplication needing to understand the details of the underlying filesystem. The VFS may be implemented in a conventional fashion bytranslating file system calls by the application into file objectmanipulations and vice versa. For example, the VFS may translate filesystem calls such as open, read, write, close, etc. into file objectcalls such as create object, delete object, etc.

The VFS may use the file name services, described elsewhere herein, totranslate file names into object identifiers. The kernel I/O driversprovide an interface to low-level object level I/O operations. Thekernel I/O drivers may be modeled after, and be similar to, Linux I/Odrivers. The layout manager may perform some of the processing on LSOtrees corresponding to files, as discussed in more detail elsewhereherein. The communication interface provides communication between theclient 104 and the servers 102. The communication interface may beimplemented using any appropriate communication mechanism. For example,if the client 104 communicates with the servers 102 via an Internetconnection, then the communication interface may use TCP/IP tofacilitate communication between the servers 102 and the client 104.

The application of FIG. 15 may correspond to the client software 124 ofFIG. 3. The VFS of FIG. 15 may correspond to one of the interfaces126-128 of FIG. 3. The file name services, kernel I/O drivers, layoutmanager, and communication interface of FIG. 15 may correspond to theserver operations software 122 of FIG. 3. Similar correlation betweencomponents of FIG. 3 and other figures may also be found.

Referring to FIG. 16, a flow chart 410 illustrates steps performed by aVFS to provide file services in connection with an application runningon the client 104. Processing begins at a first step 412 where a filesystem operation requested by an application may be translated into oneor more object operations. For example, a file operation to open a filefor reading may be converted to object operations that include obtainingan object lease for reading as discussed elsewhere herein. Following thestep 412 is a step 414 where the VFS translates the file name into anobject identifiers using the file name services discussed above inconnection with FIG. 14. Operations that follow may be performed usingthe object identifiers obtained at the step 414.

Following the step 414 is a test step 416 where it is determined if therequested operation requires the LSO tree. As discussed elsewhereherein, operations such as read, write, etc. use LSO trees correspondingto file objects. However, some possible file operations may not requireaccessing a corresponding LSO tree. If it is determined at the test step416 that the LSO tree is needed, then control transfers from the teststep 416 to a step 418 where the VFS accesses the LSO manager to performthe necessary operations. For example, for a read operation, the LSOmanager may perform processing like that illustrated in the flow chart240 of FIG. 10. Following the step 418, or following the step 416 if theLSO is not needed, is a step 422 where the operations are passed to lowlevel kernel I/O drivers (e.g., via one or more appropriate API's). Thekernel I/O drivers use the communication module to communicate betweenthe client 104 and the servers 102 in connection with performing therequested operation(s). In instances where the application running onthe client 104 has requested data and/or other information from theservers 102, the data and/or information may be passed back up throughthe communication interface, kernel I/O drivers, etc. to the VFS andultimately to the application.

Referring to FIG. 17, the client 104 is shown as having an application,file name services, user level I/O drivers, and a layout manager allprovided in user memory address space. The functionality of the VFS thatwas shown in FIG. 15 and described above may be performed instead bylibrary routines linked to the application, and thus are part of theapplication. These routines would provide functionality like thatdiscussed above in connection with FIG. 16. Accordingly, it is theapplication that uses the file name services and makes calls to the userlevel I/O drivers (like the kernel I/O drivers) and to the layoutmanager. The communication interface is still maintained in the kernelmemory address space.

Note that, for the configuration of FIG. 15, modifications are providedby modifying system processes (the operating system), which isdisadvantageous for a number of reasons. For example, if the client 104is a multiuser computing system, then modifying the operating system mayinvolve restarting the entire system and thus disrupting all of theusers. In contrast, the configuration of FIG. 17 is advantageous sinceit allows modification of the system in the application/user memoryaddress space so that the operating system of the client 104 does notneed to be modified. However, the configuration of FIG. 17 does not usea VFS, and thus does not obtain the advantageous separation of theapplication from the file system that is provided by the VFS in FIG. 15.

Referring to FIG. 18, the client 104 is shown as having an applicationin user memory address space that accesses file objects through a VFS inkernel memory address space like that illustrated in FIG. 15. However,the file name services, I/O drivers, and the layout manager all residein the user memory address space like the system illustrated in FIG. 17.The VFS communicates with components in the user memory address spacethrough a bridge between kernel memory address space and user memoryaddress space, such as a FUSE (or similar) interface. The bridge allowsfile system components to be provided in user memory space instead ofkernel address memory space while still preserving the VFS in the kerneladdress memory space. Thus, the configuration illustrated by FIG. 18provides the advantages of using a VFS, as illustrated in theconfiguration of FIG. 15, along with the advantages of having filesystem components in the user address memory space, as illustrated inthe configuration of FIG. 17.

It is possible in some instances to have applications and/or otherprocessing in the user memory address space of the client 104 accessfile objects directly, rather than through a file services layer likethe VFS and/or equivalent functionality provided by user linkablelibraries (e.g., the configuration illustrated in FIG. 17). Accessingfile objects directly may include invoking routines that create objects,read objects, modify objects, delete objects, etc. In such a case, theapplication would need to know how to interpret and/or manipulate theobject data, which may not always be desirable. For example, anapplication that accesses file objects through the VFS may not need totake into account (or even know about) the structure of an LSO treewhile an application that accesses objects directly may need to use theLSO tree. On the other hand, removing the file services layer mayprovide an opportunity for optimizations not otherwise available. Notethat, since the servers 102 exchange object information/operations withthe clients 104-106, the servers 102 may not need to distinguish or beable to distinguish between application on the clients 104-106 using afile system interface (file services like the VFS) and those that arenot.

Referring to FIG. 19, the client 104 is shown as including anapplication in the user memory address space and kernel I/O drivers, alayout manager, and file name services in the kernel memory addressspace. The configuration illustrated in FIG. 19 is like that illustratedin FIG. 15, except that the VFS is not used. In the configurationillustrated in FIG. 19, the application could directly access the filename services, the kernel I/O drivers, and the layout manager. Thecommunication interface in the kernel memory address space communicateswith the servers 102 just as in other configurations. The direct accessillustrated in FIG. 19 allows applications to manipulate file objects(via, for example, appropriate API's) while access via the VFS (orsimilar) allows applications to accesses file objects indirectly throughfile system calls to the VFS.

Referring to FIG. 20, the client 104 is shown as having an application,user level I/O drivers, a layout manager, and file name services allprovided in user memory address space. The configuration shown in FIG.20 is like that shown in FIG. 17. However, as set forth above, theconfiguration of FIG. 17 includes file service libraries that are linkedinto, and thus part of, the application. In contrast, in theconfiguration of FIG. 20, the application is not linked into librarieswith extensive file services. Instead, like the application of theconfiguration illustrated in FIG. 19, the application in theconfiguration of FIG. 20 uses minimal file services and, instead, usesand operates upon file objects directly using the user level I/Odrivers, the layout manager and, if a file name translation is needed,the file name services.

Referring to FIG. 21, the client 104 is shown as having an applicationin user memory address space and a bridge in the kernel memory addressspace. File name services, user level I/O drivers, and a layout managerare provided in user memory address space. However, unlike theconfiguration of FIG. 20, the application does not make direct calls tothe file system components in the user memory address space. Instead,the application calls the file system components indirectly through thebridge. Just as with the configuration illustrated in FIG. 18, theconfiguration of FIG. 21 advantageously locates file system componentsin the user memory address space and, at the same time, provides akernel memory address space layer between the application and the filesystem components.

Referring to FIG. 22, the client 104 is shown as having an applicationin user memory address space and a Web Services module in kernel memoryaddress space. The application may be a Web server application or anyapplication that handles communication with the Web. In an embodimentherein, the application allows communication with the client 104, whichacts as a Web server to other computing devices (not shown) that accessthe client 104 through a Web connection.

The configuration illustrated in FIG. 22 provides Web Services in amanner similar to the file services and/or file object access providedby other configurations. However, the Web Services receivesrequests/data via a Web data protocol, such as HTML, and providesresponses/data also in a Web data protocol, which may be the same ordifferent from the protocol used for requests/data. Operations handledby the Web Services may include object-level operations such as createobject, delete object, read object, modify object, modify objectmetadata, etc. It is also possible to provide more file system leveloperations, via the Web Services, that open files, read data from files,etc. by including at least some of the functionality of the fileservices, described elsewhere herein, with the Web Services. The WebServices may present to the other computing devices a conventionalwell-known Web Services protocol, such as REST or SOAP, or may provideany other appropriate protocol.

Referring to FIG. 23, the client 104 is shown as having an application,Web Services, user level I/O drivers, and a layout manager in usermemory address space. The application may include a Web connection thatallows communication with the client 104, which acts as a Web server toother computing devices (not shown) that access the client 104 throughthe Web connection. The configuration of FIG. 23 is like that of FIG. 17and FIG. 20. The advantages of the configuration shown in FIG. 23 overthe configuration shown in FIG. 22 is that, generally, changes to theconfiguration shown in FIG. 23 do not require reconfiguring kernelmemory address space processes.

Referring to FIG. 24, the 104 is shown as having an application, WebServices, user level I/O drivers, and a layout manager in user memoryaddress space. The application may include a Web connection that allowscommunication with the client 104, which acts as a Web server to othercomputing devices (not shown) that access the client 104 through the Webconnection. A bridge is provided in the kernel memory address space. Theconfiguration of FIG. 24 has similar advantages to the configurationshown in FIG. 23, but also has the advantages provided by providing thebridge, discussed elsewhere herein.

Referring to FIG. 25, the client 104 is shown as having a plurality ofapplications in user memory address space, each of which may use adifferent interface to access file objects of the servers 102. Each ofthe applications shown in FIG. 25 is meant to represent one or moreapplications. Accordingly, APP1 may present one or more applicationsthat access file objects at the servers 102 using a Web Servicesinterface. The APP1 application may include a Web connection that allowscommunication with the client 104, which acts as a Web server to othercomputing devices (not shown) that access the client 104 through the Webconnection. APP2 may represent one or more applications that access fileobjects at the servers 102 using the VFS, and APP3 may represent one ormore applications that directly operate on file objects at the servers102. The different interfaces may operate at the client 104 at the sametime.

Note that may other combinations of configurations, includingillustrated configurations, are possible so that the client 104 maysimultaneously present to applications thereon different interfaces. Forexample, it is possible to combine the configurations illustrated inFIGS. 15, 19, and 22 and/or combine the configurations of FIGS. 17, 20,and 23. Other combinations, including combinations of only twoillustrated configurations, are also possible. The servers 102 providethe file objects to the clients 104 provided: 1) the requesting clienthas appropriate authorization for whatever operation is requested forthe file objects; and 2) there is no conflict with any previous request.For example, in systems where only one client is allowed to write to anobject at any one time, the servers 102 would not allow one of theclients 104-106 to modify a particular object while another one of theclients 104-106 is also modifying the object.

Referring to FIG. 26, the servers 102 are shown in more detail asincluding one or more policy manager servers 402, one or more securitymanager servers 403, one or more audit servers 404, one or more metadataservers 405, one or more resource manager servers 406, one or more datastorage servers 407, and one or more metadata location servers 408. Eachof the servers 402-408 may be implemented as one or more unitaryprocessing devices capable of providing the functionality describedherein. For the discussion herein, reference to servers should beunderstood as a reference to one or more servers. The servers 402-408may be interconnected using any appropriate data communicationmechanism, such as TCP/IP, and may be coupled to the clients 104-106(not shown in FIG. 26) using any appropriate data communicationmechanism, such as TCP/IP.

The servers 102 may include a user management interface 412 thatfacilitates system management. The user management interface 412exchanges data with the policy management servers 402, the securitymanagement servers 403, and the audit servers 404 to affect how theservers 102 interact with the clients 104-106 and corresponding users.The data may be provided through the user management interface 412 inany one of a number of ways, including conventional interactive computerscreen input and data file input (e.g., a text file having usermanagement commands). The data may include information that correlatesclasses of users and storage parameters such as Quality of Service(QOS), RAID protection level, number and geographic location(s) ofmirrors, etc. For example, an administrator may specify through the usermanagement interface 412 that users of a particular class (usersbelonging to a particular group) store data file objects on storagedevices having a particular RAID level protection.

The servers 102 also include physical storage 414 coupled to the datastorage servers 407. Although the physical storage 414 is shown as asingle item in FIG. 26, there may be any number of separate physicalstorage units that may be geographically dispersed. In addition, theremay be different types of physical storage units having differentcapabilities. Accordingly, the physical storage 414 genericallyrepresents one or more instances of physical data storage for the systemthat is managed by the data storage servers 407, as explained in moredetail below.

Data modifications, including modifications of metadata file objectsand/or policies that affect handling/creation of metadata file objects,require appropriate security credentials. Accordingly, the securitymanager servers 403 may restrict/inhibit the ability of certainadministrators (users) to modify and/or create policies for classes ofusers.

Referring to FIG. 27, a flow chart 430 illustrates steps performed bythe user management interface 412 to obtain and use security credentialsfor accessing the policy manager servers 402. Processing begins at afirst step 432 where the user management interface 412 sends a requestto the security manager servers 403 to obtain a token (or otherappropriate security credentials) for the operation to be performed bythe user management interface 412. Following the step 432 is a test step434 where it is determined if the token has been granted (provided). Insome instances, the security manager servers 403 may not issue asecurity token at all. For example, if the administrator (user) does nothave sufficient rights to perform the desired function.

If the security token is not granted, then control passes from the step434 to a step 436 where processing is performed in connection with thesecurity token not being granted. The operations performed at the step436 may including providing a message to the administrator (user)through the security management interface 412 indicating that theadministrator does not have sufficient rights to perform the desiredoperation. Following the step 436, processing is complete.

If it is determined at the test step 434 that a security token has beengranted (provided) by the security manager servers 403, then controlpasses from the test step 434 to a step 438 where the user managementinterface 412 provides the security token, and user id information, tothe policy manager servers 402. Of course, information indicating thedesired operation/modification may also be provided at the step 438.Following the step 438 is a test step 442 where it is determined if thepolicy manager servers 402 have allowed the requested operation. Notethat, in some instances, the policy manager servers 402 may not allow aparticular operation even though the security manager servers 403 haveprovided a security token. For example, if the user id and the userindicated by the security token do not match and/or if the requestedoperation and the operation indicated by the security token do notmatch.

If it is determined at the test step 442 that the requested operation isnot allowed, then control passes from the test step 442 to the step 436,described above, where processing is performed to indicate that thereare security issues. The processing performed at the step 436 mayinclude providing a message to an administrator (user) indicating thatthe operation cannot be performed because of insufficient securityrights. The message provided when the step 436 is reached from the step442 may be different than the message provided when the step 436 isreached from the step 434.

If it is determined at the test step 442 that the requested operation isallowed, then control passes from the test step 442 to a step 444 wherethe operation is performed. Performing the operation at the step 444 mayinclude modifying policy data, as described in more detail elsewhereherein. Following the step 444, processing is complete.

Thus, an administrator (user) accessing the policy manager servers 402would first provide identification information to the security managerservers 403 that would return a security token (perhaps having anexpiration time). The administrator presents the token andidentification information to the policy manager servers 402, whichwould decide to grant or deny access based on the token and theidentification information. Note that the security mechanism illustratedby the flow chart 430 of FIG. 27 may be extended to be used inconnection with accessing any of the servers 402-408 and/or other data.For example, one of the clients 104-106 could obtain/modify file objectsby first requesting a security token from the security manager servers403 prior to performing an operation that includes operations with fileobjects. Accordingly, for the discussion herein, it can be assumed thataccess to file objects, servers, etc. includes appropriate securityprocedures like those illustrated in FIG. 27.

The policy manager servers 402 handle placement and protection of fileobjects. An administrator (user) may input, through the user managementinterface 412, different policy templates that may be assigned todifferent ones of the clients 104-106, different users, differentclasses of users, or any other appropriate group. For example, a policytemplate may indicate that, for a particular group of users, whenever anew file is created, a mirror will be created that is geographicallyfarther from the initial data set by at least a certain distance. Insuch a case, when a first user of the group creates an initial data setin New York, a mirror may be automatically created in Los Angeles while,when a second user creates an initial data set in Los Angeles, a mirrormay be created in New York.

The audit servers 404 may be used to provide system auditing capability.A user may communicate to the audit servers 404 through the usermanagement interface 412. The user may indicate the type of informationto be audited (tracked).

The resource manager servers 406 keep track of available systemresources. In some instances, the resource manager servers 406 mayinteract with the policy manager servers 402 in connection withestablishing policy templates and/or assigning policy templates. In somecases, a user may attempt to construct a policy template that isimpossible to fulfill if assigned to a group. For example, if all of thephysical data storage is in a single geographic location, then it wouldnot be appropriate to have a policy template indicating that new filesshould include a mirror that is geographically distant from the initialdata set.

The resource manager servers 406 receive information from othercomponents of the system in order to be able to keep track whichresources are available. Whenever a resource is added to the system, theresource or another component reports that information to the resourcemanager servers 406. For example, if new physical storage is added tothe system, the new physical storage itself, or a corresponding one ofthe data storage servers 407, sends a message to the resource managerservers 406. Similarly, if a resource becomes full (e.g., a physicaldisk is full) or is removed from the system (planned removal orunplanned resource failure), information is provided to the resourcemanager servers 406. In an embodiment herein, system resources maycorrespond to portions of the physical storage 414 and/or data servers407 that manage the physical storage 414.

Referring to FIG. 28, a resource table 460 is shown as including aplurality of entries 462-464, each of which corresponds to a particularstorage resource. Although only three entries are shown, the table 460may contain any number of entries. The table 460 may be implementedusing any appropriate technique, including an array, linked list, etc.

Each of the entries 462-464 includes a resource field identifying aparticular resource corresponding to the entry. In an embodiment herein,each of the entries 462-464 may correspond to a particular one of thedata storage servers 407 and/or a portion thereof. Each of the entries462-464 includes a status field corresponding to the status of thecorresponding resource. In an embodiment herein, the status field mayindicate that a resource is on-line (available) or off-line(unavailable). The status field may also indicate the percentage of usedspace of a resource, and perhaps indicate any performance degradation.

Each of the entries 462-464 may also include a capabilities field thatindicates the capabilities of the corresponding resource. In anembodiment herein, when the resources represent storage areas, thecapabilities field may indicate particular capabilities of acorresponding storage area. Particular capabilities may include theresource being green (low energy use through, for example, spinningdisks down when not in use), capable of data deduplication (maintainingonly a single copy of data that is otherwise duplicated), capable ofvarious RAID configurations, etc. The capabilities field may indicateany appropriate data storage capabilities.

Referring to FIG. 29, a flow chart 480 indicates operation of theresource manager servers 406 in connection with maintaining informationabout system resources. Processing begins at a first step 482 where theresource manager servers 406 are initialized with information aboutresources. The initialization processing performed at the step 482 maytake any form, including loading a fixed table of initially availableresources, having the resource manager servers 406 poll systemresources, etc.

Following the step 482 is a test step 484 where the resource managerservers 406 wait for new information to be provided. In an embodimentherein, after initialization, the resource manager servers 406 wait toreceive information from other system components. In other embodiments,it may be possible to have the resource manager servers 406 periodicallypoll system components to see if anything has changed. If it isdetermined at the test step 484 that no new information is available,control loops back on the test step 484 to continue polling.

Once it is determined at the test step 484 that new information isavailable, then control transfers from the test step 484 to a test step486 where it is determined if the new information relates to a newresource added to the system. If so, then control transfers from thetest step 486 to a step 488 where the new entry is added to the resourcetable that is managed by the resource manager servers 406. Following thestep 488, control transfers back to the step 484 to continue waiting fornew information.

If it is determined at the step 486 that the received resourceinformation does not related to a new resource (and thus relates to achange of an existing resource), then control transfers from the step486 to a step 492 where the existing entry is located in the resourcetable. Following the step 492 is a test step 494 where it is determinedif the capability is being changed for the modified resource. Thecapability of a resource may change under many different circumstances.For example, a resource may degrade and lose capabilities, a resourcemay be modified/enhanced and gain capabilities, a local manager of aresource may decide to make certain capabilities available/unavailable,etc.

If it is determined at the step 494 that the capabilities of a resourcehave changed, then control transfers from the test step 494 to a step496 to change the capabilities field for the resource being modified.Otherwise, control transfers from the test step 494 to a step 498 tochange the status field of the resource being modified (e.g., resourceis full, resource is off-line, resource is on-line, etc.). Followingeither the step 496 or the step 498, control transfer back to the step484, discussed above, for another iteration.

Note that the resource manager servers 406 may represent a plurality ofseparate computing devices that may be dispersed throughout the system.Furthermore, each of the separate computing devices may maintain its owncopy of the resource table. The separate computing devices that are usedto implement the resource manager servers 406 may or may not shareresource information and may or may not receive the same resource statusmessages. In instances where information sharing and/or receipt ofstatus messages is not perfect, then each of the computing devices mayhave a somewhat different version of the resource table and it ispossible for no one version of the resource table to reflect acompletely accurate picture of the exact state of all of the resourcesof the system.

The physical storage 414 may be provided using relatively inexpensiveoff-the-shelf mass produced storage hardware. In an embodiment herein,at least some of the physical storage 414 may be implemented usingserial ATA disk drives, which are available from a number ofmanufactures such as Seagate and Western Digital. As discussed elsewhereherein, the physical storage may be geographically dispersed. However,each portion of the physical storage may be managed/controlled by atleast one of the data storage servers 407, which may be implementedusing conventional computing devices local to the corresponding portionof the physical storage 414.

In an embodiment herein, the data storage servers 407 may present an OSDStandard interface to the system. Thus, the servers 102 and/or theclients 104-106 may access physical storage 414 through the data storageservers 407 using OSD calls and may receive information/data accordingto the OSD protocol. In addition, the data storage servers 407 mayhandle managing/posting the capabilities and status of differentportions of the physical storage 414. Thus, for example, when a portionof the physical storage 414 is managed by a particular server of thedata storage servers 407, the particular server may send a message tothe resource manager servers 406 indicating the new status.

Referring to FIG. 30, a flow chart 510 illustrates steps performed bythe resource manager servers 406 in connection with servicing an inquiryfor a resource with particular capabilities (i.e., finding a resourcewith particular capabilities). Processing begins at a first step 512where a pointer, used to iterate through each entry of the resourcetable, is set to point to the first entry. Following the step 512 is atest step 514 where it is determined if the pointer points past the endof the table (i.e., all entries have been examined). If so, then controlpasses from the test step 514 to a step 516 where a result indicating nomatch for the requested capabilities is returned by the resource managerservers 406. Following the step 516, processing is complete.

If it is determined at the test step 514 that the pointer used toiterate through the entries does not point past the end of the table,then control transfers from the test step 514 to a test step 518 whereit is determined if the entry currently indicated by the pointer is amatch for the requested capability. Note that the test at the step 518may include checking the status of a resource to ensure that theresource is on-line and not full or otherwise unusable. If it isdetermined at the step 518 that the resource indicated by the pointerhas the requested capability, then control transfers from the test step518 to a step 522 where the resource manager servers 406 return anindicator indicating the matching resource. Following the step 522,processing is complete.

If it is determined at the step 518 that the resource indicated by thepointer does not have the requested capability (or is off-line, full,etc.), then control transfers from the test step 518 to a step 524 wherethe pointer is incremented. Following the step 524, control transfersback to the step 514, discussed above, for another iteration.

The system described herein may be used with any server, or any group ofservers, capable of providing file objects to clients. The particularform of the file objects may vary without departing from the spirit andscope of the invention. In some instances, the order of steps in theflow charts may be modified, where appropriate. The system describedherein may be implemented using a computer program product/softwareprovided in a computer-readable storage medium.

While the invention has been disclosed in connection with variousembodiments, modifications thereon will be readily apparent to thoseskilled in the art. Accordingly, the spirit and scope of the inventionis set forth in the following claims.

1. A method of tracking storage resources, comprising: providing a tablecontaining storage resources along with capabilities and statusesthereof; updating the table in response to a change of status of astorage resource; updating the table in response to a change incapabilities of a storage resource; and in response to an inquiry for astorage resource having a particular capability, searching the table fora storage resource having the particular capability.
 2. A method,according to claim 1, further comprising: adding an element to the tablein response to a new resource being added to the system.
 3. A method,according to claim 1, wherein the capabilities include RAID striping,data deduplication, and green operation.
 4. A method, according to claim1, wherein the status is one of: on-line, off-line, and full.
 5. Amethod, according to claim 1, wherein the storage resources are diskdrives.
 6. A method, according to claim 5, wherein the disk drives aremanaged by data storage servers that present an OSD interface for thedisk drives.
 7. A method, according to claim 1, wherein the table ismaintained by a resource manager server that receives information aboutstorage resources from other servers.
 8. Computer software, provided ina computer-readable storage medium, that tracks storage resources,comprising: a table that contains storage resources along withcapabilities and statuses thereof; executable code that updates thetable in response to a change of status of a storage resource;executable code that updates the table in response to a change incapabilities of a storage resource; and executable code that searchesthe table for a storage resource having a particular capability inresponse to an inquiry for a storage resource having the particularcapability.
 9. Computer software, according to claim 8, furthercomprising: executable code that adds an element to the table inresponse to a new resource being added to the system.
 10. Computersoftware, according to claim 8, wherein the capabilities include RAIDstriping, data deduplication, and green operation.
 11. Computersoftware, according to claim 8, wherein the status is one of: on-line,off-line, and full.
 12. Computer software, according to claim 8, whereinthe storage resources are disk drives.
 13. Computer software, accordingto claim 12, wherein the disk drives are managed by data storage serversthat present an OSD interface for the disk drives.
 14. Computersoftware, according to claim 8, wherein the table is maintained by aresource manager server that receives information about storageresources from other servers.
 15. A resource manager that managesstorage resources for a storage system, comprising: a processing device;and computer-readable memory coupled to the processing device, thecomputer-readable memory having a table provided in a data structure andcontaining storage resources along with capabilities and statusesthereof, the computer-readable memory also having executable code thatupdates the table in response to a change of status of a storageresource, executable code that updates the table in response to a changein capabilities of a storage resource, and executable code that searchesthe table for a storage resource having a particular capability inresponse to an inquiry for a storage resource having the particularcapability.
 16. A resource manager, according to claim 15, wherein thecomputer-readable memory also contains executable code that adds anelement to the table in response to a new resource being added to thesystem.
 17. A resource manager, according to claim 15, wherein thecapabilities include RAID striping, data deduplication, and greenoperation.
 18. A resource manager, according to claim 15, wherein thestatus is one of: on-line, off-line, and full.
 19. A resource manager,according to claim 15, wherein the storage resources are disk drives.20. A resource manager, according to claim 19, wherein the disk drivesare managed by data storage servers that present an OSD interface forthe disk drives.
 21. A data storage system, comprising: a plurality ofclients; and a plurality of servers coupled to the clients, wherein asubset of the servers manage storage resources using a table containingstorage resources along with capabilities and statuses thereof, whereinthe subset updates the table in response to a change of status of astorage resource, updates the table in response to a change incapabilities of a storage resource, and searches the table for a storageresource having the particular capability in response to an inquiry fora storage resource having a particular capability.
 22. A data storagesystem, according to claim 21, wherein the subset of servers adds anelement to the table in response to a new resource being added to thesystem.
 23. A data storage system, according to claim 21, wherein thestorage resources are disk drives.
 24. A method of providing informationto a resource manager of a data storage system, comprising: providinginformation to the resource manager in response to a change incapabilities of a storage resource; providing information to theresource manager in response to a change in status of a storageresource; and providing information to the resource manager in responseto adding a new storage resource.
 25. A method, according to claim 24,wherein the storage resources are disk drives.